MPI Debugging with Handle Introspection

The Message Passing Interface, MPI, is the standard programming model for high performance computing clusters. However, debugging applications on large scale clusters is difficult. The widely used Message Queue Dumping interface enables inspection of message queue state but there is no general interface for extracting information from MPI objects such as communicators. A developer can debug the MPI library as if it was part of the application, but this exposes an unneeded level of detail. The Tools Working Group in the MPI Forum has proposed a specification for MPI Handle Introspection. It defines a standard interface that lets debuggers extract information from MPI objects. Extracted information is then presented to the developer, in a human readable format. The interface is designed to be independent of MPI implementations and debuggers. In this paper, we describe our support for introspection in the TotalView debugger and test it against a reference introspection implementation in Open MPI. We also describe how the debugger interfaces with the MPI implementation.